Jan 7, 2014 4:05am
Re: the captures of my site are so sparse
The crawl-delay of "30" means that a crawler has to wait 30 seconds before crawling a page.
Let's say it crawls you front page, from that front page it finds other pages that are linked to from the front page. A crawler then wants to crawl those pages, the pages linked to from the front page that was crawled.
But the robots.txt files says "Crawl-delay: 30" that means a crawler needs to wait 30 seconds before crawling an other page linked to from the front page.
Hope I explained it well... :)
I will make a quick copy of your whole site for the wayback machine later today, I'll keep you informed...
You do not need to essentially have a robots.txt page.
If you don't have a robots.txt page a crawler can just download from the site whatever it wants.
This post was modified by Arkiver on 2014-01-07 12:05:47